Coby Meyers 
Amy Proger 
American Institutes for Research 


Yasuyo Abe 
Phyllis Weinstock 
Vincent Chan 
IMPAQ International 


In collaboration with the Midwest Beating the Odds Research Alliance 


Many states are attempting to identify schools that perform better than schools 
with similar populations. Such “beating-the-odds” schools offer opportunities to 
identify promising practices that can be implemented by other schools serving similar 
populations. This study uses data from the Michigan Department of Education to 
demonstrate how methodological decisions affect which schools are identified as 
beating the odds. Policymakers should carefully consider the methodology choices 
made to identify beating-the-odds schools to ensure that the lessons learned come 
from schools that really do perform better than expected. 


This brief summarizes the findings of Abe, Y., Weinstock, P., Chan, V., Meyers, C., Gerdeman, R. 
G., & Brandt, W. C. (2015). How methodology decisions affect the variability of schools identified as 
beating the odds (REL 2015-071). Washington, DC: U.S. Department of Education, Institute of 
Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional 
Educational Laboratory Midwest. That report is available at: http://ies.ed.gov/ncee/edlabs/projects/ 
project.asp?ProjectID=353. 
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Why this study? 


Identifying and examining schools that exceed achievement expectations—sometimes referred to as 
schools that “beat the odds”—are part of efforts to implement strategies that support continuous improve- 
ment and school turnaround. Some state and local education agencies identify beating-the-odds schools to 
recognize them with awards and to motivate similar schools, especially schools that serve disproportionate- 
ly high-needs students and that exhibit lower performance. The higher performing schools may be studied 
to develop or identify effective strategies and interventions for supporting and transforming low-performing 
schools. For example, Arizona (Waits et al., 2006), Delaware (Grusenmeyer, Fifield, Murphy, Nian, & Qian, 
2010), and New York City (Connell, 1999) have or have had processes to identify and learn from beating- 
the-odds schools. 


Like other state education agencies, the Michigan Department of Education is interested in identifying pol- 
icies and practices that distinguish beating-the-odds schools from their counterparts and in implementing 
these policies and practices in struggling schools serving similar student populations. Michigan uses two 
statistical methods to identify beating-the-odds schools: a prediction method and a comparison method. 
The prediction method identifies a school as beating the odds if its actual level of performance is higher 
than its predicted level of performance given school demographics. The comparison method identifies a 
school as beating the odds if it outperforms other demographically similar schools. These two methods 
often produce different results. In school years 2009/10 and 2010/11, less than a third of schools identified 
as beating the odds by either of the two methods individually were identified by both methods. And within 
each method, the set of schools changed from one year to the next, with fewer than half the schools consis- 
tently identified as beating the odds for two consecutive years. 


A group of Michigan Department of Education and school district stakeholders interested in the processes 
used to learn from beating-the-odds schools came together in 2012 as the Beating the Odds Research 
Alliance. This group has worked with Regional Educational Laboratory Midwest to further examine the 
inconsistencies produced by the two statistical methods. The study used data from school and student 
assessment and demographic records from the Michigan Department of Education for K-12 public schools, 
covering school years 2007/08—2010/11. The analyses document how the set of schools identified as beating 
the odds differs across statistical methods (prediction method or comparison method) and specifications 
(different performance measures, school characteristics, and school sample configurations; box 1). 


Although this study focused on the needs identified by a research alliance in a single state, the challeng- 
es of identifying beating-the-odds schools are not unique to the state. This study provides information 
that can assist other states and districts in developing or revising their technical approaches to identifying 
schools that exceed performance expectations and in understanding the potential limitations. 


Box 1. Types of statistical models used for identifying beating-the-odds schools examined in the 
report 


For both prediction and comparison methods the study team compared the baseline model (the same specifi- 
cations that the Michigan Department of Education used in its 2010/11 analyses) to several alternative models 
(specifications that used different performance measures, school characteristics, or school sample configu- 
rations). The study team compared the schools identified as beating the odds using these specifications and 
documented the degree of consistency between the methods. The baseline and alternative specifications are 
presented in box table 1. 


Table 1. Baseline and alternative specifications used in the study 


Type of 
specification 


Baseline specifications 
for prediction method? 


Baseline specifications for 
comparison method?” 


Alternative specifications 


Performance Michigan Top-to-Bottom percentile | Michigan Top-to-Bottom percentile | Composite academic achievement 
measure ranking ranking index developed by the study team 
School - Percentage of students who are + Percentage of students who are + Percentage of students who are 


characteristics 


English learner students 


* Percentage of students who are 


eligible for free or reduced-price 
lunch 


+ Percentage of students who are 


racial/ethnic minority students 


+ Percentage of students with 


disabilities 


English learner students 


+ Percentage of students who are 


eligible for free or reduced-price 
lunch 


+ Percentage of students who are 


racial/ethnic minority students 


+ Percentage of students with 


disabilities 


* School configuration 

+ Locale 

+ Total enrollment 

+ Special education center status 
* State foundation allowance 


English learner students 


* Percentage of students who are 


eligible for free or reduced-price 
lunch 


+ Percentage of students who are 


racial/ethnic minority students 


+ Percentage of students with 


disabilities 


+ School configuration 

+ Locale 

+ Total enrollment 

+ Magnet school indicator 

+ Percentage of students who are 


female 


School sample 
configuration 


The sample includes schools 
serving all grade levels. 


The sample includes schools 
serving all grade levels. 


Schools are identified separately for 
each level (elementary, middle, high). 


a. Top-to-Bottom percentile ranking is based on a school composite index that aggregates measures of school achievement, 
including average standardized test scores and graduation rates, changes in average standardized test scores and graduation 
rates over time, and the achievement gap between the top 30 percent of students and the bottom 30 percent of students. This 
Top-to-Bottom percentile ranking is available only for 2010/11. The difference in characteristics used by method was determined 


by the Michigan Department of Education. 


b. Baseline school characteristics for the comparison method are used to identify groups of demographically similar schools. 


The model specification for the baseline and alternative analyses are described in box table 2. Model A rep- 
resents the baseline model established by the Michigan Department of Education for each method. Models B, 
C, and D represent alternative models, each changing one of the following from the baseline: the performance 
measure, school characteristics, or school configuration. Model E applies all three alternatives from the base- 
line at the same time. Model F includes alternative school characteristics and sample configurations but uses 
the same performance measures as the Michigan Department of Education. 


(continued) 


Box 1. Types of statistical models used for identifying beating-the-odds schools examined in the 
report (continued) 


Table 2. Model specifications for the baseline and alternative analyses used in the study 


Model A Model B Model C Model D 


Alternative 
performance 


Alternative measure, Alternative 
Data and sample performance Alternative Alternative characteristics, characteristics 
specification choice Baseline measure characteristics configuration and configuration and configuration 


Performance measures 
Top-to-Bottom ranking percentile® v v v Vv 


Alternative composite index based 
on math/reading scale scores v v 


School characteristics 


Baseline (different across methods) v v v 


Alternative? 
(comparable across methods) v v v 


School sample configurations 
All levels pooled together v v Vv 
By school level® v Vv Vv 


a. Available only for 2010/11. 


b. Alternative school characteristics were selected based on a series of stepwise multivariate regressions on the alternative 
composite performance measure, starting with the baseline school characteristic variables originally applied by Michigan in both 
methods. Inputs were selected if they were significant in three out of the four years of data. 


c. Identifies beating-the-odds schools separately for elementary, middle, and high school grades. 


Source: Authors’ analysis based on data from school and student assessment and demographic records from the Michigan De- 
partment of Education for K-12 public schools, covering school years 2007/08-2010/11. 


What the analyses say about identifying schools that are beating the odds 


The study team compared the schools identified as beating the odds using two methods (prediction and 
comparison) and a variety of specifications that altered performance measures, school characteristics, and 
school sample configurations. The study team found that different methods and model specifications iden- 
tified different sets of beating-the-odds schools. The study team also examined how the group of schools 
identified as beating the odds changed across four consecutive school years for each method and found that 
few schools were identified as beating the odds in more than one year. More detailed findings are described 
in the following sections. 


The prediction method and comparison method identified different sets of beating-the-odds schools 


The first analyses for the study compared the schools identified as beating the odds using the prediction 
and comparison methods. Reanalyzing the 2010/11 data using the same set of schools and specifications as 
the Michigan Department of Education (model A; see box 1), the study team identified 75 beating-the-odds 
schools using the prediction method and 28 using the comparison method, with 39 percent agreement 
(table 1).! Under the baseline model the prediction and comparison methods use different school char- 
acteristics. Using an alternative approach with performance measures, school characteristics, and school 
sample configurations different from those in the baseline model (model F; see box 1), the study team iden- 
tified 71 beating-the-odds schools using the prediction method and 35 using the comparison method, with 
32 percent agreement (see table 1). The alternative approach uses the same school characteristics for both 
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Table 1. Beating-the-odds school identification results varied by prediction and comparison 
methods, 2010/11 


Number of beating the odds schools identified by 


Agreement 
rate between 
Prediction Comparison Both methods 
Statistical model method method methods (percent) 
Baseline specifications (model A) 75 28 20 39 
Alternative school characteristics and configurations (model F) 71 35 17 32 


Note: The total number of schools in the school year 2010/11 study sample is 3,563. For model A the sample includes 2,888 
schools for the prediction method and 2,791 schools for the comparison method. For model F the sample includes 2,888 schools 
for both methods. 


Source: Authors’ analysis based on data from school and student assessment and demographic records from the Michigan De- 
partment of Education for K-12 public schools. 


methods. Regardless of the specification, agreement rates between methods for schools identified as beating 
the odds were relatively low, suggesting that choice of method—rather than specification—explains the 
differences in the sets of schools identified as beating the odds. 


The choice of school performance measures and school characteristics greatly influenced which schools were 
identified as beating the odds 


The second analyses for the study compared the schools identified as beating the odds across different spec- 
ifications. For both the comparison and the prediction methods, the study team calculated the agreement 
rate between the baseline model and three alternative specifications (table 2). The analyses showed that 
using different school performance measures greatly influenced which schools were identified as beating the 
odds and that using different school characteristics also influenced which schools were identified as beating 
the odds, although to a lesser extent. Using different school sample configurations did not influence which 
schools were identified as beating the odds. 


The largest differences in the sets of schools identified as beating the odds occurred when performance 
measures were altered (model B). For the prediction method, switching from the baseline specification for 
the performance measure to the alternative performance measure reduced the number of schools identi- 
fied as beating the odds by half, from 75 to 37, and the resulting agreement rate was only 11 percent (see 
table 2). For the comparison method, switching from the baseline to the alternative performance measure 
more than doubled the number of schools identified, from 28 to 70, and the resulting agreement rate was 
18 percent. 


Differences in the sets of schools identified as beating the odds also emerged when the alternative spec- 
ification for school characteristics was used (model C), though to a lesser degree (see table 2). The sets 
of schools generated using the baseline school characteristics employed by the Michigan Department of 
Education and the group of common school characteristics constructed by the study team contained a 
similar number of schools. The results show a 75 percent agreement rate using the prediction method and a 
41 percent agreement rate using the comparison method. 


Finally, few differences in the identified schools emerged when the alternative specification for school 
sample configuration (model D) was used (see table 2). In fact, for the prediction method, there was 
100 percent agreement between the baseline and alternative models. 


Table 2. Using different performance measures caused the largest difference in the sets of 
schools identified as beating the odds among the three alternative specifications, 2010/11 


Model A Model B Coxe (=) I eg Model D 


Alternative Alternative Alternative 
performance school school 
Baseline model measure characteristics configuration 


Prediction method 


Number of schools identified as beating the odds 75 37 71 75 
Number of schools that overlapped with baseline model na 6 55 75 
Agreement rate with baseline model (percent) na 41. 75 100 


Comparison method 


Number of schools identified as beating the odds 28 70 30 30 
Number of schools that overlapped with baseline model na 9 12 26 
Agreement rate with baseline model (percent) na 18 41 90 


na is not applicable. 


Note: The total number of schools in the 2010/11 study sample is 3,563. For the prediction method the sample includes 2,888 
schools for model A, 3,300 schools for model B, 2,887 schools for model C, and 2,888 schools for model D. For the compari- 
son method the sample includes 2,791 schools for model A, 3,231 schools for model B, 2,887 schools for model C, and 2,888 
schools for model D. 


Source: Authors’ analysis based on data from school and student assessment and demographic records from the Michigan De- 
partment of Education for K-12 public schools. 


The results indicate that using different school performance measures and school characteristics led to 
different sets of schools being identified as beating the odds, with school performance measures being par- 
ticularly influential. Moreover, the extent to which different performance measures or school character- 
istics affect the sets of schools identified as beating the odds may depend on the method used to identify 
these schools. For example, the lower level of agreement for the comparison method (41 percent compared 
with 75 percent for the prediction method) for model C suggests that the choice of school characteristics 
might be especially important for an approach that seeks to identify demographically similar schools for 
comparison. 


Fewer than half the schools were identified as beating the odds in more than one year 


Agreement rates for schools identified in consecutive years (using the same method) averaged 48 percent 
for the prediction method and 49 percent for the comparison method. These agreement rates are derived 
by applying the same specifications (model E) to each method across all years, suggesting that year-to-year 
variation in the schools identified as beating the odds reflects changes in the underlying school perfor 
mance and characteristic data or statistical noise (random error) rather than changes in specifications. 


An examination of how frequently individual schools were identified as beating the odds found that, 
among the schools identified at least once, only 9 percent were identified in all four years using the predic- 
tion method (16 of 186) or the comparison method (15 of 175). Overall, more than half the schools that 
were identified as beating the odds at least once by either method in any year were not identified again in 
any of the three other years analyzed for this study. This indicates that relative school performance levels, 
as measured in this study, change across years and contribute to inconsistency in sets of schools identified 
as beating the odds across consecutive years. 


Implications for research and policy 


Recognizing schools that are beating the odds requires careful consideration by policymakers because of 
the many possible approaches to identifying such schools. This study does not recommend one method 
or one group of technical specifications over another, but rather was designed to inform the process of 
developing and evaluating a technical approach to identifying beating-the-odds schools. The findings show 
that different statistical methods or technical specifications of performance measures, school characteris- 
tics, and school sample configuration result in inconsistent sets of Michigan schools identified as beating 
the odds. Moreover, the sets of beating-the-odds schools change across years, even when using the same 
method and technical specification. 


Such variability in sets of beating-the-odds schools presents challenges when states or districts wish to 
showcase schools that are outperforming expectations, especially when the intention is to share policies 
and practices of those schools with their counterparts. Depending on the stakes involved, states and dis- 
tricts intending to identify schools that are outperforming their peers may want to consider developing 
more or less stringent criteria—or applying several models—as the basis of identifying such schools. 


Ultimately, states and districts will want to align the technical specifications for identifying beating-the- 
odds schools with policy goals and decisions. For example, states that recently have adopted more rigorous 
standards will want to use performance measures that are aligned to those standards rather than older 
performance measures that were not so aligned. 


Limitations of the study 


This study is limited in several respects. 


First, the study explored only a limited group of methods and specifications; there may be alternative 
methods and specifications that produce additional variation in schools identified as beating the odds. 
Future studies might explore whether sets of schools identified as beating the odds vary when alternative 
methods and specifications are included. 


Second, the study used agreement rate as the primary measure to assess variation across sets of schools 
identified as beating the odds. The study did not examine how close schools were to being identified under 
different methods or model specifications. 


Third, the study examined only variation that arose when different methods, model specifications, and 
years were examined. However, the choices in method, model specifications, and year also affect which 
schools could be included in the analyses. That is, if a school is missing a particular data element that is 
used in one of the analyses, it would have been excluded from the sample. Thus, different samples of schools 
may have been included in analyses due to missing data, thereby making it possible that agreement rates 
presented in this report are lower than is really the case. The samples of schools varied in sizes between 
O percent and 3.5 percent across methods (within model), between 0 percent and 12.5 percent across model 
specifications, and between 5.7 percent and 8 percent across years (within models).” 


These limitations should be considered in interpreting the results and applying the model specifications 
described here to future efforts to identify beating-the-odds schools. 


Notes 


To measure the differences and similarities between two sets of schools identified as beating the odds, 
the study team calculated an adjusted agreement rate, which is the ratio of the number of schools that 
appear in both sets to the average number of schools in a set. The adjusted agreement rate ranges from 
O percent to 100 percent. It attains the maximum possible value when the two sets have exactly the 
same number of schools and exactly the same schools. 

See table B3 in appendix B of Abe et al. (2015) for sample sizes. 
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the widespread dissemination of the results of research and evaluation throughout the United States. 
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